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EVALUATING ESSENTIALS TRAINING: 



INTRODUCTION 



Countries that begin implementing the Intel® Teach to the Future Essentials course after March of 2006 
are required to collect data using the Intel® Teach Essentials End of Training Survey (see Appendix 
A) to help support program improvement. This guide will help new evaluators understand how to use 
and analyze the data from this required survey. In addition to this guide, EDC is available to talk with 
Education Managers and local evaluators to review and support the development of local evaluation 
plans. 

This End of Training evaluation toolkit provides guidelines on: 

• End of Training Survey administration, 

• The evaluation objectives of the End of Training Survey, 

• Organizing the survey data, specifically how to clean and code the data, and 

• Analyzing the survey data 



Many countries also conduct larger evaluations of this program, and EDC has an Optional Toolkit of other 
protocols for both End of Training and Impact evaluations that may be of interest to those evaluators. 
EDC strongly encourages use of the Optional Toolkit protocols for gathering more information to get a 
richer picture of how trainings are playing out in different countries. 

The guidelines in this toolkit are designed to provide an evaluation framework for local evaluators and 
Education Managers who are in the initial phase of evaluating Intel® Teach to the Future. This toolkit 
will also help clarify the types of analyses that can be done with the End of Training data for those 
Education Managers and local evaluators who have been collecting End of Training data for many years. 

EDC has tried to ensure that all guidelines and instructions included in this toolkit are stated in a simple 
and easy-to-follow manner. This toolkit is meant to help even those without educational evaluation 
experience to administer the survey and analyze the data. Education Managers and local evaluators can 
also contact EDC if they have additional questions or concerns. 
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Survey administration 

The survey should be administered to all participants on the last day of the training. Each country will 
need to decide how to administer the survey depending upon their particular needs and challenges. 

Some countries use on-line surveys and some use paper surveys. Country Education Managers and their 
external evaluation agencies will need to decide what method is most appropriate based on program 
resources, staff, and technology available at training sites. 

The purpose of conducting the End of Training Survey is to find out if the localized version of the 
Essentials Program is being delivered effectively, and to provide local program staff with the information 
necessary to improve and adapt the program to meet the needs of teachers in their country. To collect 
the best data, the survey should be administered to as diverse and non-biased a sample as possible. EDC 
recommends that participants at all trainings be given the opportunity to fill out the End of Training 
Survey. If it is not possible to administer the survey at all trainings, the sample of trainings chosen to be 
surveyed should be representative of the trainings occurring in the country as a whole. For example, if 
there are trainings happening in rural and urban areas, then both urban and rural area trainings need to 
be included in the sample. Likewise, if teachers from privately-funded and government-funded schools 
are participated in the training, then teachers from both of these groups should be sampled in roughly 
the same proportion as they are represented in the program population as a whole. 
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EVALUATING ESSENTIALSTRAINING: 



EVALUATION OBjECTIVES 



What can we learn from the End of Training data? 

The End of Training data capture participants' perceptions of the Essentials course, and reveal what 
the training experience was like for them. These kind of data provide valuable information about 
what is working and not working in the delivery of the training, and can be used to inform program 
improvement. For example, finding that a large number of participants do not feel very prepared to 
implement what they learned in their classrooms would suggest there is a problem with the training. It 
could be that the trainers need better training themselves, or that the participants need more instruction 
on certain program concepts that are new to them. Discovering gaps between the intended delivery 
of the training and participants' experience of the training is vital if program staff want to identify 
emerging problems and improve the training experience. 

The End of Training Survey asks sets of questions about five key aspects of the program and about the 
technology infrastructure in participants' schools. The infrastructure information can help evaluators 
and program staff understand how well the teachers will be able to implement what they learn in the 
training. Each of the sets of questions is described below. 

Focus of the training [Items Q1 a, b, c, and d.] 

This section asks teachers if they felt the training focused on teaching technology and practical 
classroom examples, and if it provided a collaborative learning environment. This information can help 
evaluators and program staff understand if the training and related materials localized to their country 
are successfully communicating the key messages of the Intel® Teach to the Future Essentials course to 
participants (see Figure 1). 
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Figure 1 

Ql. To what extent do the following statements describe the Intel® Teach to the Future training in which you participated? 
For each item below, select the choice that best represents your experience. 





Not 


Small 


Moderate 


Great 




At All 


Extent 


Extent 


Extent 


a) Focused on integration of technology into your teaching. 










b) Provided useful teaching strategies to use with your stu- 
dents. 










c) Illustrated effective uses of technology with students. 










d) Provided opportunities to collaborate with other educators 
during training. 











Effectiveness of the training [Items Q<? a, b, c, d, and e] 

The second set of questions asks participants to report how prepared they feel to implement the 
instructional- and technology-related practices covered in the training. Responses to this set of 
questions indicate how effective the training was in preparing teachers to implement later in the 
classroom. These data can help program staff understand if the training program is achieving its goal 
of preparing teachers to use instructional approaches modeled in the training (items Q2 a and e) and to 
integrate technology into their teaching (items Q2 b, c, and d). See Figure 2. 



Figure 2 

Q2. Having completed your training, how well prepared do you feel to do the following activities with your students? For 
each item below, select the choice that best represents your experience. 





Not At All 
Prepared 


Somewhat 

Prepared 


Moderately 

Prepared 


Very Well 
Prepared 


a) Implement methods of teaching that emphasize independent 
work by students. 










b) Integrate technology into your teaching. 










c) Support your students in using technology in their school- 
work. 










d) Evaluate technology-based work your students produce. 










e) Align your teaching and assessments with required curricu- 
lum content. 
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Effectiveness of the trainer [Items Q3 o, b and c] 

The third set of questions asks teachers to rate how successful their trainers were at leading them 
through the instructional- and technology-related practices covered in the training. Responses to these 
questions can help evaluators and program staff see whether trainers are adequately prepared to deliver 
the Essentials training (see Figure 3). 



Figure 3 

Q3. Think about the trainer who led your workshop. For the items below, select the choice that best reflects your experience. 





Not At All 


Somewhat 


Adequately 


Very 


a) How successful was he/she at leading participants through 
the process of creating unit plans? 










b) How successful was he/she at engaging the group in discus- 
sions of pedagogical and classroom management issues? 










c) How successful was he/she at guiding participants through 
the technology skills covered in the training? 











Usefulness of the training components [Items Q4 a, b, c, d, e,f and g and Q5] 

The fourth set of questions investigates teacher perceptions about the usefulness of the instructional 
concepts (Q4 a and b) and hands-on activities (Q4 c, d, e, f, and g and Q5) covered in the training. The 
questions on the instructional content (Q4 a and b) ask teachers how useful they found material on 
Essential and Unit Questions and the discussions around instructional topics. Responses to these questions 
can help evaluators and program staff gauge how successful the training was in helping participants see the 
relevance of project-based approaches to their teaching practice. Teachers from countries where project-based 
instruction is unfamiliar may have more difficulty seeing the relevance of these strategies. In such cases, 
the training curriculum may need extra material showing how project-based instruction can support student 
learning. 

The remaining questions in this set (Q4 c, d, e, f, and g and Q5) ask teachers if they found activities such 
as creating student publications/multimedia presentations/web sites and teacher support resources useful. 
These activities all use technology in some form, and teacher responses to these items give evaluators and 
program staff insight into the relative value participants place on the various individual modules, and on 
various types of modules (see Figure 4). 
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Figure 4 



Q4. How useful was each of the following components of the training in helping you learn how to integrate technology into 
your teaching practices? For each item below, select the choice that best reflects your experience. 





Not 

Useful 


Somewhat 

Useful 


Moderately 

Useful 


Very 

Useful 


a) Creating, and exploring the uses of Essential Questions and 
Unit Questions. 










b) Discussing and thinking through the pedagogical topics. 










c) Locating and evaluating resources for your unit. 










d) Creating student multimedia presentations. 










e) Creating student publications. 










f) Creating teacher support materials. 










g) Creating student web sites. 











Q5. Will the ideas and skills you learned from the Intel® Teach to the Future training help you successfully integrate tech- 
nology into your students’ activities? 

□ Definitely Not 

□ Probably Not 

□ Probably Yes 

□ Definitely Yes 



Participant background [QB, and Q7 and Q8] 

The fifth set of questions asks participants if they trained to be a Master Teacher (MT) or a Participant 
Teacher (PT), and then asks about their years of teaching experience, and their familiarity with 
integrating technology into their teaching before the training. This information is helpful for 
understanding who the program is reaching, and also allows investigation of whether a participants' prior 
experience with technology influences their experience in the Essentials course. 

The MT/PT ratio indicates whether the program is reaching the expected number of Master and 
Participant Teachers. If the number of Participant Teachers is lower than expected, this may suggest that 
Master Teachers are having difficulty recruiting participants or securing facilities for delivering trainings. 
This is information program staff would certainly need to know. 
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Responses to the remaining participant background questions provide important information about 
participants' teaching experience and familiarity with technology integration. This information can 
provide insight into the kinds of teachers that are either interested in, or are being recruited for, the 
training (see Figure 5). 

The results from these items can be used to divide the data by teaching experience or by technology 
experience to examine if different types of participants benefit differently from the training. This type 
of analysis generally requires a statistical software program, such as SPSS. For evaluators who do not 
have SPSS software, instructions for basic analyses that can be run in Microsoft Excel are included in the 
Analzying the Data section of this guide. For evaluators who have SPSS software, instructions for analyses 
that can be done with SPSS are covered in the SPSS for Analysis section of this guide. 

Figure 5 

Q6. Which course of the Intel® Teach to the Future program have you just completed? 

□ Master Teacher Training 

□ Participant Teacher Training 

Q7. How many years of teaching experience do you have? 

□ Less than 3 

□ 3 to 9 

□ 10 to 20 

□ Over 20 

Q8. Please select the term that best describes your level of experience integrating technology into your teaching BEFORE 
the Intel® Teach to the Future training. 

□ You have never used technology with your students (Never Used) 

□ You have tried a few times to have your students use technology during your classes (New User) 

□ You have a few lessons involving technology that you feel comfortable having your students do during your classes (Interme- 
diate User) 

□ You regularly have your students use technology to engage in school work (Advanced User) 

□ You are a technology leader in your school, or you train others in the use of technology (Expert User) 
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Infrastructure [Item Q9, Q10, Q11, and Q12] 

This final set of questions asks the Participant Teachers about their access to technology resources 
and the Internet in their school. Each teacher's access to technology infrastructure will strongly 
influence whether and how they are able to follow up on the Essentials course. This information can 
tell the evaluator if the program is targeting schools with sufficient infrastructure to support teachers' 
integration of technology after the training. If the data reveal that participants do not have adequate 
infrastructure, program staff will need to decide whether to make changes to the training or change 
the program's recruitment approach. This information can also help Ministries of Education staff make 
informed decisions about technology infrastructure policy (see Figure 6). 

Figure 6 

Q9. In your school where do you have access to computers? 

□ No access to computers in my school 

□ Classroom and in the computer lab or media center 

□ Only in the classroom 

□ Only in a computer lab or media center 
Q10. How many computers are in your classroom? 

□ 0 computers 

□ 1 computer 

□ 2-4 computers 

□ 5-7 computers 

□ More than 7 computers 

Qll. In your school where do you have access to the internet? 

□ No access to computers in my school 

□ Classroom and in the computer lab or media center 

□ Only in the classroom 

□ Only in a computer lab or media center 

Q12. How easy or difficult is it to schedule time in the computer lab/media center? 

□ Very difficult 

□ Difficult 

□ Easy 

□ Very easy 

□ Do not know 
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Adding localized questions 

EDC recommends that evaluators add localized questions to help program staff better understand how 
the trainings are being delivered and received in their country. The localized questions can collect data 
about the context of the trainings, participant demographics, and country-specific characteristics of the 
program. However, questions in a survey need to follow a logical order, so care should be taken when 
deciding where to insert the questions. 
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IMk 

J EVALUATING ESSENTIALSTRAINING: 



□ RCjANIZINCj THE DATA 



The following sections explain how to use Microsoft® Excel to create a database with the survey results, 
and then how to use Excel to do simple statistics to analyze the results. Since, Excel is limited in the 
level of statistics that it can do, suggestions on how to use SPSS (a Statistical Package for the Social 
Sciences) are also included. However, evaluators should use the statistical and other software packages 
they are familiar with to create and analyze databases. 

Data coding 

If the evaluators have distributed paper copies of the survey, the responses need to be coded into 
numbered responses and entered into whichever database is being used. The coding sheet is provided 
in Appendix B. If the evaluators are using an on-line survey, they need to assign the responses the 
appropriate numerical value so that the dataset will be equivalent to the dataset from other countries. 
Lastly, the year and quarter when the data were collected should be recorded in the spreadsheet. 

Data entry procedure 

If paper surveys were administered, the codes need to be entered manually into an Excel spreadsheet 
or data file of whichever data analysis software package the evaluator is using. If an online survey was 
administered and the responses were assigned the appropriate numerical values, the data will be in a 
data file or Excel spreadsheet already. If not, please refer to above section on "Data coding". Depending 
on how the program stores the online data collected, the evaluator may need to copy and paste the 
data into an Excel spread sheet. It is important to remember that data from each survey respondent 
constitute a row in the Excel spreadsheet or whichever data file the evaluator is using. 

EDC provides an Excel file that contains the appropriate database structure for summary data (percentage 
only); this is available for download at http://cct.edc.org/teachfuture.asp. Evaluators can also get the 
Excel spreadsheet from the country Education Manager. 

Data cleaning procedure 

After the End of Training data have been collected, the next step is to clean the data prior to analyzing 
it. The following section offers step-by-step instructions on how to clean End of Training data in 
preparation for analysis. It is assumed that the data are in an Excel spreadsheet. If not, please refer to 
above section on "Data entry procedure". 
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Step 1: Ensuring dataset is correct 

Before anything is done with the data, the evaluator should make sure the End of Training data in the 
dataset come from the correct version of the survey. A quick way to check is to look at the numbering 
scheme and length of the items. The most recent (2006) version of the End of Training Survey ranges 
from Q1 through Q12 and fills 27 columns of data (see Picture 1). The year and quarter that the data 
come from should be specified at the top of the spreadsheet. 



Picture 1 : Making sure data is right version. 
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Step 2: Assigning ID numbers 

Before any types of data manipulation are done, the evaluator should make sure each record has an ID 
number. This will allow the evaluator to sort the data back to its original state in case a problem or 
confusion emerges. 

To add an ID number: 

1) Put the cursor on the cell labeled "ql" 

2) Go to the Insert Menu at the top of the screen then scroll down and click on the Columns tab. 
This should insert a blank column to the left of the ql column. Give the new column a title like 
ID number or Participant ID. 

To add ID numbers in the column: 

1) Put the number 1 in the top cell corresponding to the first participant. 

2) Put a number 2 in the cell corresponding to the second person, and a 3 corresponding to the 
third participant. 

3) Highlight all three cells; move the cursor to the bottom right hand corner of the three cells and 
a small black plus sign (+) will appear. Click and drag the plus sign all the way down to the last 
participant. This will create ID Numbers for each participant (see Picture 2). 



3 icture 2: Assigning ID numbers. 
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Step 3: Cleaning/checking the dataset 

The evaluator should check to ensure that all values in the Excel dataset are valid response options. If a 
question has response options that range from 1- 4, for example, there should not be any other values in 
this column. In addition, the evaluator should make sure that there are only numbers, and no text, in 
the response columns. 



End of Training Required Survey for Essentials 
14 





EVALUATING ESSENTIALSTRAINING: 



ANALYZING THE DATA 



Running frequencies in Excel 

This section gives instructions for computing frequencies in Excel for all of the questions in the survey. 
These instructions may be hard to memorize, so it is recommended that the evaluator print out the 
instructions before use. They will need to be followed exactly. 

Step 1: Data preparation 

There should be at least 5 blank rows below the last row of participant data. 

Go five cells below the last record in the column of data corresponding to ql and type the numbers 1,2, 
3, and 4 in numerical order, one cell below the next (see Picture 3). 



Picture 3: Illustration of Step 1. 
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Step 2: Calculating response counts 

2a) Skip a cell and type in exactly (see Picture 4): 

=FREQUENCY ( 



Picture 4: Illustration oFStep 2a. 
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2b) Next, highlight the column from the first cell of participant data to the last cell of participant data, 
add a comma, and highlight the 1, 2, 3, 4, column that was made, then close the parenthesis and press 
ENTER (see Picture 5). 



Picture 5: Illustration of Step 2b. 
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2c) After pressing ENTER, that cell should present a value. Click in that same cell (if it is not already 
clicked) hold down the SHIFT key and press the down arrow three times so it highlights four cells 
(including the one with the value). 

Then (with all four cells highlighted) press F2, then press CTRL, SHIFT, and ENTER (at the same time). 
This will calculate the number of times that 1, 2, 3, and 4 appeared in that column (see Picture 6). In 
other words, this is a count of the number of times each response option appeared in the data for a 
specified question. This can then be used to get proportion counts. 



Picture B: Illustration of Step 2c. 
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Step 3: Calculating proportions 

Step 2 produced counts of the values for that column. To get percentages in Excel, the next step is 
to calculate the proportion of responses falling under each value. Percentages and proportions are 
calculated through Excel formulas. When cell numbers are inserted in the formula. Excel includes the 
value of that cell number into the formula. The actual cell(s) need to be clicked on or highlighted for 
Excel to recognize and insert cell numbers into the formula (this applies specifically to steps 3b, 3c, 

3d, and 3e that follow). The following sub-steps (3a through 3e) need to be completed to calculate the 
proportions, and the following instructions detail which cells need to be clicked on or highlighted within 
the formula. 

3a) Move one cell down below the Step 2 value counts and type exactly: 

= SUM ( 

Then highlight the new counts from step 2c (see Picture 7, cells B318 to B321 , for an example), close 
parenthesis and press ENTER. This will give a grand total of all responses for that column. 

Picture 7: Illustration of Step 3a. 
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3b) Next, individually divide the counts of 1, 2, 3, and 4 by the grand count total. This will need to be 
done separately for each value. So, move down one cell and type in the formula (keep in mind that the 
following instruction details which cells need to be clicked on or highlighted within the formula): 



=(the cell corresponding to the l's counts) /(the grand total of responses for the column) 



and press ENTER (see Picture 8). This should move the cursor down one cell. If this does not happen 
automatically, move down one cell. This will calculate the proportion of 'l's in responses to Question la. 



Picture 8: Illustration oFStep 3b. 
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3c) Move down one cell and type in (keep in mind that the following instruction details which cells need 
to be clicked on or highlighted within the formula): 



=(the cell corresponding to the 2's counts) /(the grand total of responses for the column) 

and press ENTER (see Picture 9). This should move the cursor down one cell. If not, move down one cell. 
This will calculate the proportion of '2's in responses to Question la. 



Picture 9: Illustration of Step 3c. 
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3d) Move down one cell and type in (keep in mind that the following instruction details which cells need 
to be clicked on or highlighted within the formula): 



=(the cell corresponding to the 3's counts) /(the grand total of responses for the column) 



and press ENTER (see Picture 10). This should move the cursor down one cell, if not move down one cell. 



Picture 10: Illustration of Step 3d. 
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3e) Move down one cell and type in (keep in mind that the following instruction details which cells need 
to be clicked on or highlighted within the formula): 



= (the cell corresponding to the 4's counts) /the grand total of responses for the column) 



and press ENTER (see Picture 11). This should move the cursor down one cell, if not move down one cell. 
Picture 11: Illustration of Step 3e. 




3f) Highlight the four new cells created by steps 3b, 3c, 3d, and 3e (See Picture 11, cells B325 to B328, 
represent the four cells in this example), move the cursor up to the FORMAT button on the menu and 
click - > CELLS then under category go to percentage. This will give the frequency percentages. 
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Step 4: Verifying the percentage 

To verify that the spreadsheet is displaying the correct percentage, add up the percentages to be sure 
that they equal 100%. To do this move down one cell lower and type exactly 

= SUM ( 

Highlight the four percentage columns that were just created, close parenthesis and press ENTER. If 
correct, that cell should read 100% (see Picture 12). 



Picture 12: Illustration of Step 4. 
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Step 5: Calculating frequencies for remainder of data 

5a) Lastly, instead of repeating the above process for every item, there is an easy way to calculate the 
remaining frequencies once the calculations for one item (column) have been completed. First, highlight 
everything new that was just entered (from the cell containing the 1 to the cell containing the 100%), 
then move the mouse to the bottom right corner of the values highlighted until a small cross appears 
(see Picture 13). 



Picture 13: Illustration of Step 5a. 
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5b) Click and drag to the right to the last item column. When the mouse is released, the frequencies and 
checks of all the items should appear (see Picture 14). 



Picture 14: Illustration of Step 5b. 
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Step B: Transferring the calculated frequencies 

EDC recommends that the percentages of each response option for each question be copied into another 
EDC provided summary data sheet (shown in picture 15) for ease of interpretation of results. EDC 
provides this spreadsheet available for download at [http://cct.edc.org/teachfuture.asp]. 



Picture 15: Illustration of Summary Data Sheet. 
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EVALUATING ESSENTIALS TRAINING: 



INTERPRETING THE DATA 



The End of Training Survey is designed to provide Education Managers and other program staff with 
regular, frequent feedback on their programs, so that they can understand whether the program is 
meeting the needs and expectations of the educators it was created to serve. After entering the End of 
Training Survey data into Excel and calculating the frequencies. Education Managers and local evaluators 
need to then review and interpret the findings. Below are descriptions of some of the ways that the 
survey frequencies can be examined to provide useful information about the program. 

Meeting internal program goals and expectations 

All Education Managers set internal goals and expectations for the program at different stages of 
implementation and program maturity. The End of Training Survey data can be used to measure whether 
the program is meeting those goals and expectations. The different items on the survey are designed 
to provide information about different program components, and general information about program 
participants. The Evaluation Objectives section of this toolkit describes how each set of items can be 
interpreted on its own. In addition to those interpretations, evaluators and program managers can also 
look at multiple sets of items on the End of Training Survey to understand more about important aspects 
of the program. A few of the key aspects about which the End of Training survey can provide feedback 
are: 

• Recruitment: Participant responses to Questions 8-12 can help Education Managers understand 
what kinds of teachers are being recruited for the training. Is the program recruiting the 
appropriate population of teachers? Do they have technology resources available or prior 
experience? For example, if there is an expectation that the program will only recruit teachers 
with some access to technology, and some technology experience, responses to these questions 
will reveal whether recruitment efforts are targeting the intended participants. 

• Curriculum localization: Each Education Manager is charged with ensuring that the Intel® Teach 
Essentials curriculum is localized by education experts in their country so that the program is 
relevant and meaningful to teachers in that country. Participants' responses to Questions 2 and 
4 specifically can provide insight into how useful participants feel the training was and whether 
they believe they can use what they learn in the training in their teaching. 

• Master Teacher training: The success of the program relies on the quality of the Master Teacher 
training. The End of Training Survey provides information about the quality of the Master 
Teacher training in two ways. First, the survey is administered to Master Teachers, and their 
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responses reveal what they thought of the training. In addition. Participant Teachers' responses 
to Questions 1 and 3 also reveal whether the training experience adequately communicated the 
core messages of the program, and whether participants found their trainers to be effective. 

Meeting program-wide benchmarks 

Working in conjunction with EDC evaluators, Intel Teach program staff have established benchmarks for 
specific items on the End of Training Survey that can help local program staff better understand how 
the program is developing in their country. Benchmarks were established by analyzing survey responses 
from participants in a number of different countries over time, and determining the percent of positive 
responses programs could reasonably be expected to receive if they are offering a high quality program. 
Programs that are not yet meeting benchmarks should consider themselves to still in development. These 
programs might want to focus their evaluation efforts on improving the localization and identifying 
challenges and obstacles that participants encounter. Programs that are meeting the benchmarks 
would be considered to be mature programs. Their evaluation efforts might focus on documenting the 
program's impact. The benchmarks selected by Intel program staff for the specific survey items are listed 
below: 

• Question la: 89% of teacher respondents indicate the training focused on integration of 
technology into their curriculum to a "great extent" or "moderate extent." 

• Question lb: 81% of teacher respondents indicate the training provided teaching strategies to 
apply with their student to a "great extent" or "moderate extent.". 

• Question lc: 86% of teacher respondents indicate the training illustrated effective uses of 
technology with students to a "great extent" or "moderate extent." 

• Question 2a: 80% of teacher respondents indicate they are "very well" or "moderately" prepared 
to implement teachings that emphasize independent work by students. 

• Question 2b: 85% of teacher respondents indicate they are "very well" or "moderately" prepared 
to integrate educational technology into the grade or subject they teach. 

• Question 2c: 82% of teacher respondents indicate they are prepared to support their students in 
using technology in their schoolwork. 

If participant responses do not meet these benchmarks, it may indicate that the program is not 
successfully communicating its core messages to participants, or that the curriculum has not been 
localized enough for teachers to see clearly how to apply what they learn in the training to their 
classroom teaching. 

Looking at results over time 

As program evaluators collect End of Training data each quarter, it can be useful to keep track of program 
participants' responses to the program over time. Charting the data longitudinally enables evaluators 
and program staff to gauge whether participant responses become more positive as the program matures. 
If program staff make substantial changes to the implementation process, the curriculum, the Master 
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Teacher training or other aspects of the program, they can see whether there are corresponding changes 
in participant feedback from the surveys administered at the time those changes were made. 

Deeper analyses that EDC commonly does on Training Data 

The sections below explain some of the key questions that we have used as a first look at analyzing the 
data. Some examples of interesting questions to ask of the data that can be answered using crosstabs are 
included below. 

Communicating important messages: Is the training helping teachers think about practical strate- 
gies for technology integration into their classroom? 

Items Q2 a, b, and c ask participants if they felt that the training was focused on practical and useful 
strategies to integrate technology into their teaching. If the results indicate that most participants 
do not think that the training is focused on these issues, this may indicate that the training is not 
effectively communicating the core material to the teachers. 

Do teachers with different backgrounds experience the training differently? Is the program more effective 
for experienced teachers or less experienced teachers? For example, if you examine the results on Q2 a, 
b, and c by the level of technology experience, this would allow you to understand if teachers with more 
technology experience see the training as more focused on practical integration strategies than less 
experienced teachers. 

Post training confidence: Do teachers leave the training with a sense of preparation to implement 
new activities back in their schools? 

The results on Q3 a, b, c, d, and e indicate the level of preparedness of the participants after the 
training. Some of the questions are about technology issues (b, c, and d) and some are about using new 
teaching strategies in the classroom (a, and e). Are teacher more prepared on one set of items than on 
the other? Does this vary by their level of technology experience? 

Utility of the training topics: Do participants feel that the topics covered help them reach the 
program's goals? 

Q5 asks participants about a number of key topics and activities. Some of the items ask about supporting 
student technology use (Q5 d, e, f, and g) and other items ask about new teaching strategies (Q5 a, b, 
and c). Do teachers' perceptions vary by level of experience with teaching or experience with technology? 



End of Training Required Survey for Essentials 
29 




End of Training Required Survey for Essentials 

30 





EVALUATING ESSENTIALSTRAINING: 



SPSS FOR ANALYSIS 



Running frequencies and crosstabs in SPSS 

SPSS is a statistical and data management package that provides a broad range of capabilities for the 
entire data analysis process. Analyzing the End of Training data does not require advanced statistics; 
but SPSS does offer the option of doing some deeper analyses with the data than simple frequencies, 
such as cross tabulations (often abbreviated as crosstab). A frequency describes the distribution of one 
variable, while a crosstab can describe the distribution of two or more variables simultaneously. Each cell 
in a crosstab table shows the number of respondents that gave a specific combination of responses. See 
Picture 16 a for examples of a frequency distribution and a crosstab. 



Picture 1 6a: Illustration of Frequency Distribution and Crosstab. 
Frequencies 



la. Focused on integration of technology into your teaching 





Frequency 


Percent 


Valid Percent 


Cumulative 

Percent 


Valid Not At All 


2618 


1.2 


1.2 


1.2 


Small Extent 


15950 


7.0 


7.0 


8.2 


Moderate Extent 


59128 


26.1 


26.1 


34.3 


Great Extent 


149014 


65.7 


65.7 


100.0 


Total 


226710 


99.9 


100.0 




Missing System 


181 


.1 






Total 


226891 


100.0 







Crosstabs 



6. Which course of the Intel Teach to the Future program have you just completed? * la. Focused on integration of technology into your teaching 

Crosstabulation 









la. Focused on integration of technology into your 
teaching 














Moderate 












Not At All 


Small Extent 


Extent 


Great Extent 


Total 


6. Which course of 


MasterTeacherTraining 


Count 


283 


1349 


5032 


14781 


21445 


the Intel Teach to the 
Future program have 
you just completed? 




% within 6. Which course 
of the Intel Teach to the 
Future program have you 
just completed? 


1.3% 


6.3% 


23.5% 


68.9% 


100.0% 




Participant Teacher 


Count 


2271 


14248 


53579 


133669 


203767 




Training 


% within 6. Which course 
of the Intel Teach to the 
Future program have you 
just completed? 


1.1% 


7.0% 


26.3% 


65.6% 


100.0% 


Total 




Count 


2554 


15597 


58611 


148450 


225212 






% within 6. Which course 
of the Intel Teach to the 
Future program have you 
just completed? 


1.1% 


6.9% 


26.0% 


65.9% 


100.0% 
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To obtain frequencies, open the SPSS data file and from the menu choose: 

Analyze — > 

Descriptive Statistics — > 

Frequencies 

Then move the questions for which frequencies are needed into the variable box by clicking on the 
question and then clicking on the arrow button in the middle of both boxes (see Picture 16 b). The 
results open in a separate Output window, and give the frequencies for all variables chosen (refer back to 
Picture 16 a). 

Picture 1 Bb: Illustration of Frequencies Procedure. 
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To obtain crosstabs, open the SPSS data file and from the menu choose: 

Analyze — > 

Descriptive Statistics — > 

Crosstabs 

Crosstabs will create a 2-dimensional table. Move the variable to be displayed as the row variable to 
Row(s): and move the variable to be displayed as the column variable to Column (s): (see Picture 16c). 




Picture 1 Be: Illustration of Crosstabs Procedure. 



At the very bottom of the crosstabs window there are three buttons: "Statistics", "Cells", and "Format". 
Clicking on each of these options opens up more specific choices of crosstab analyses. For the purposes of 
the End of Training data analysis, the guide will only discuss the "Cells" button to give percentages within 
each cell in the crosstabs table. If the box next to "Row" is checked it will give percentages within the row 
variable. Likewise, if the box next to "Column" is checked, it will give percentages for the column variable 
(see Picture 16d). 

Picture 1 Bd: Illustration oF Crosstabs Procedure. 
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A 2 x 2 crosstab could look at whether participants with varying prior experience with technology 
differ in how useful they find the training module that requires them to create student web sites. An 
evaluator can use a third variable to layer the crosstabs output. To continue with the above example, an 
evaluator could also look at the crosstabs within another variable such as the training they participated 
in (Participant Teacher Training or Master Teacher Training). If the training variable were added to the 
Layer box (under the Row(s): and Column(s): boxes) then one 2x2 (prior experience by usefulness 
of creating student web sites) crosstabs table would be generated for PTs and another 2x2 (prior 
experience by usefulness of creating student web sites) crosstabs table would be generated for MTs. 
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Appendix A 




End of Training Required Survey 



Focus of the Training 

Please mark the right response box to the questions below. 



1. To what extent do the following statements describe the Intel® Teach to the Future training in 
which you participated? For each item below, select the choice that best represents your 
experience. 





Not 
At All 


Small 

Extent 


Moderate 

Extent 


Great 

Extent 


a) Focused on integration of technology 
into your teaching. 










b) Provided useful teaching strategies to 
use with your students. 










c) Illustrated effective uses of technology 
with students. 










d) Provided opportunities to collaborate 
with other educators during training. 











Effectiveness of the Training 

Please mark the right response box to the questions below. 



2. Having completed your training, how well prepared do you feel to do the following activities 
with your students? For each item below, select the choice that best represents your experience. 





Not At All 
Prepared 


Somewhat 

Prepared 


Moderately 

Prepared 


Very Well 
Prepared 


a) Implement methods of teaching that 
emphasize independent work by 
students. 










b) Integrate technology into your 
teaching. 










c) Support your students in using 
technology in their schoolwork. 










d) Evaluate technology-based work your 
students produce. 










e) Align your teaching and assessments 
with required curriculum content. 
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Effectiveness of the Trainer 

Please mark the right response box to the questions below. 



3. Think about the trainer who led your workshop. For the items below, select the choice that 
best reflects your experience. 





Not At All 


Somewhat 


Adequately 


Very 


a) How successful was he/she at leading 
participants through the process of 
creating unit plans? 










b) How successful was he/she at 
engaging the group in discussions of 
pedagogical and classroom management 
issues? 










c) How successful was he/she at guiding 
participants through the technology skills 
covered in the training? 











Usefulness of Training Components 

Please mark the right response box to the questions below. 



4. How useful was each of the following components of the training in helping you learn how to 
integrate technology into your teaching practices? For each item below, select the choice that 
best reflects your experience. 





Not 

Useful 


Somewhat 

Useful 


Moderately 

Useful 


Very 

Useful 


a) Creating, and exploring the uses of 
Essential Questions and Unit Questions. 










b) Discussing and thinking through the 
pedagogical topics. 










c) Locating and evaluating resources for 
your unit. 










d) Creating student multimedia 
presentations. 










e) Creating student publications. 










f) Creating teacher support materials. 










g) Creating student web sites. 
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5. Will the ideas and skills you learned from the Intel® Teach to the Future training help you 
successfully integrate technology into your students’ activities? 

□ Definitely Not 

□ Probably Not 

□ Probably Yes 

□ Definitely Yes 



Participant Background 

Please mark the right response box to the questions below. 

6. Which course of the Intel® Teach to the Future program have you just completed? 

□ Master Teacher Training 

□ Participant Teacher Training 

7. How many years of teaching experience do you have? 

□ Less than 3 

U 3 to 9 

□ 10 to 20 

□ Over 20 



8. Please select the term that best describes your level of experience integrating technology into 
your teaching BEFORE the Intel® Teach to the Future training. 

□ You have never used technology with your students (Never Used) 

□ You have tried a few times to have your students use technology during your 
classes (New User) 

□ You have a few lessons involving technology that you feel comfortable having 
your students do during your classes (Intermediate User) 

□ You regularly have your students use technology to engage in school work 
(Advanced User) 

□ You are a technology leader in your school, or you train others in the use of 
technology (Expert User) 



Infrastructure 

Please mark the right response box to the questions below. 

9. In your school where do you have access to computers? 

□ No access to computers in my school 

□ Classroom and in the computer lab or media center 

□ Only in the classroom 

□ Only in a computer lab or media center 
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10. How many computers are in your classroom? 

□ 0 computers 

□ 1 computer 

□ 2-4 computers 

□ 5-7 computers 

□ More than 7 computers 

1 1 . In your school where do you have access to the internet? 

□ No access to computers in my school 

□ Classroom and in the computer lab or media center 

□ Only in the classroom 

□ Only in a computer lab or media center 

12. How easy or difficult is it to schedule time in the computer lab/media center? 

□ Very difficult 

□ Difficult 

□ Easy 

□ Very easy 

□ Do not know 



4 




Appendix B 



Intel® 2006 End of Training Survey Coding Sheet 

This sheet provides a coding key to the response options (or possible answers) to each 
question. The number provided against each response option is the number that should be 
entered into your data file. 



Focus of the Training 

Please mark the right response box to the questions below. 



1. To what extent do the following statements describe the Intel® Teach to the Future training in 
which you participated? For each item below, select the choice that best represents your 
experience. 





Not 
At All 

11] 


Small 

Extent 

[2] 


Moderate 

Extent 

[3] 


Great 

Extent 

[4] 


a) Focused on integration of technology 
into your teaching. 










b) Provided useful teaching strategies to 
use with your students. 










c) Illustrated effective uses of technology 
with students. 










d) Provided opportunities to collaborate 
with other educators during training. 











Effectiveness of the Training 

Please mark the right response box to the questions below. 



2. Having completed your training, how well prepared do you feel to do the following activities 
with your students? For each item below, select the choice that best represents your experience. 





Not At All 
Prepared 

[1] 


Somewhat 

Prepared 

[2] 


Moderately 

Prepared 

[3] 


Very Well 
Prepared 

[4] 


a) Implement methods of teaching that 
emphasize independent work by 
students. 










b) Integrate technology into your 
teaching. 










c) Support your students in using 
technology in their schoolwork. 










d) Evaluate technology-based work your 
students produce. 










e) Align your teaching and assessments 
with required curriculum content. 
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Effectiveness of the Trainer 

Please mark the right response box to the questions below. 



3. Think about the trainer who led your workshop. For the items below, select the choice that 
best reflects your experience. 





Not At All 

[ 1 ] 


Somewhat 

[ 2 ] 


Adequately 

[3] 


Very 

[41 


a) How successful was he/she at leading 
participants through the process of 
creating unit plans? 










b) How successful was he/she at 
engaging the group in discussions of 
pedagogical and classroom management 
issues? 










c) How successful was he/she at guiding 
participants through the technology skills 
covered in the training? 











Usefulness of Training Components 

Please mark the right response box to the questions below. 



4. How useful was each of the following components of the training in helping you learn how to 
integrate technology into your teaching practices? For each item below, select the choice that 
best reflects your experience. 





Not 

Useful 

[ 1 1 


Somewhat 

Useful 

[2] 


Moderately 

Useful 

[3] 


Very 

Useful 

[4] 


a) Creating, and exploring the uses of 
Essential Questions and Unit Questions. 










b) Discussing and thinking through the 
pedagogical topics. 










c) Locating and evaluating resources for 
your unit. 










d) Creating student multimedia 
presentations. 










e) Creating student publications. 










f) Creating teacher support materials. 










g) Creating student web sites. 
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5. Will the ideas and skills you learned from the Intel® Teach to the Future training help you 
successfully integrate technology into your students’ activities? 

□ Definitely Not [1] 

□ Probably Not [2] 

□ Probably Yes [3] 

□ Definitely Yes [ 4 ] 



Participant Background 

Please mark the right response box to the questions below. 



6. Which course of the Intel® Teach to the Future program have you just completed? 

□ Master Teacher Training [1 ] 

□ Participant Teacher Training [2] 



7. How many years of teaching experience do you have? 

□ Less than 3 [1] 

□ 3 to 9 [2] 

□ 10 to 20 [3] 

□ Over 20 [ 4 ] 



8. Please select the term that best describes your level of experience integrating technology into 

your teaching BEFORE the Intel® Teach to the Future training. 

□ You have never used technology with your students (Never Used) [1] 

□ You have tried a few times to have your students use technology during your 
classes (New User) [2] 

□ You have a few lessons involving technology that you feel comfortable having 
your students do during your classes (Intermediate User) [3] 

□ You regularly have your students use technology to engage in school work 
(Advanced User) [ 4 ] 

□ You are a technology leader in your school, or you train others in the use of 
technology (Expert User) [ 5 ] 

Infrastructure 

Please mark the right response box to the questions below. 

9. How many computers are in your classroom? 

□ 0 computers [1] 

□ 1 computer [2] 

□ 2-4 computers [3] 

□ 5-7 computers [ 4 ] 

□ More than 7 computers [5] 
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10. In your school where do you have access to computers? 

□ No access to computers in my school [1] 

□ Classroom and in the computer lab or media center [2] 

□ Only in the classroom [3] 

□ Only in a computer lab or media center [4] 

1 1 . In your school where do you have access to the internet? 

□ No access to computers in my school [1] 

□ Classroom and in the computer lab or media center [2] 

□ Only in the classroom [3] 

□ Only in a computer lab or media center [4] 

12. How easy or difficult is it to schedule time in the computer lab/media center? 

□ Very difficult [1] 

□ Difficult [2] 

□ Easy [3] 

□ Very easy [4] 

□ Do not know [5] 
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